WIP: discover kernel-source roots at runtime, and replace the two LLM source tiers with one agent review - #1263

…te table.

Source resolution had two model tiers. The per-kernel fallback picked a path
from a grep shortlist for any kernel above 5% of GPU time; the whole-table pass
audited the finished resolution table. The first was the weaker of the two and
mostly redundant: it fired only where the second already looked, at a higher
threshold, and it saw less -- a symbol and eight paths, without the launcher
frames the table pass gets. Neither could see what the kernel actually is,
because both were tool-free completions fed a prompt assembled in advance.

Both are gone. In their place one agent session audits the candidate table
itself, and is handed paths rather than contents: the raw table, the resolution
audit, the TraceLens report and per-category metrics, the model directory, the
framework source roots. It reads what the evidence leads it to instead of what
was guessed to be relevant ahead of time, so it can open a file and confirm it
defines the kernel it is credited with rather than ranking paths by keyword.

The pass is mandatory on the agent route and ungated: the thresholds only ever
described what a completion was worth, and the deterministic route keeps its
own no-LLM guarantee by not running the stage at all. It retries once, and a
definitive failure records an error-severity trace-health warning and leaves the
deterministic table standing -- losing the audit costs some candidates, while
failing the run would cost the hours of benchmarking behind the trace.

Three bounds keep the added freedom from becoming a new failure mode:

- Proposals only. The session may revise where a kernel lives and whether it is
  worth dispatching; it may not touch what the trace measured. Derived state is
  recomputed by the deterministic stamping pass afterwards, so
  classify_patchability stays the one gate rather than gaining a model-written
  second one, and a restrictive routability hint is honoured while a permissive
  one is not.
- A revised path must resolve under a known framework root, and a candidate
  already resolved by the active finder is not overridable: reading the tree
  cannot beat knowing which symbol the binary exports.
- The session may run shell commands, so the candidate source files are
  fingerprinted around it and a review that modified them is discarded. The
  benchmark that follows would otherwise measure an unrecorded edit.

Artifacts are now kernel_candidates.raw.json (deterministic),
kernel_candidates.json (reviewed) and kernel_candidates_revisions.json (what
changed and why), so a bad dispatch can be traced to the stage that caused it.

Verified against a recorded MiniMax-M3 run: the deterministic stage is
unchanged at 6 of 16 routable covering 13.74% of GPU time.

WIP: the new module's own test suite is still to come.
Co-authored-by: Cursor <cursoragent@cursor.com>

The batch filter drops a candidate below the GPU-share floor and says so in its
log, then discards the reason. When the orchestrator had named that kernel, the
handler canonicalized the id and fell through to the validation guards, whose
failure result was recorded as an attempt. The 9.47% kernel of the last 8k/1k
run therefore shows two attempts and two failures against a backend that never
ran, and reads in the report as a technical failure when the cause was a
threshold. UNATTEMPTED_BELOW_MIN_GPU_PCT already exists to say this properly.

The filter now hands its skip reasons back, and a named kernel dropped for one
that means "no backend ran" -- below the floor, its group exhausted, merged into
an op-fanout sibling -- returns skipped with that reason. The clean skipped exit
existed already but sat behind an ``elif`` only reachable when no kernel was
named, which is the one case that cannot happen when the batch queue is empty
and the orchestrator is picking targets by hand.

Two identity fixes come with it, and have to: the ledger keys on kernel_id plus
source_file, and the empty-candidate branch never passed the source through even
though it had the full candidate list in hand. Every rejection was filed under
an empty source, which is also why the retry quota went to an empty-string
bucket and the two failures never blocked redispatch. Backfilling the source
without the change above would have turned that harmless accident into a real
block, since a single recorded attempt exhausts the default cap of one -- so the
guards now carry the resolved source, and the branch backfills the candidate it
already loaded.

Verified against the recorded run: naming the gate-rejected kernel returns
status=skipped with reason=below_min_gpu_pct instead of a failure, and a guard
rejection now carries its source_file.

Co-authored-by: Cursor <cursoragent@cursor.com>

The review is handed a tool-enabled session because the deterministic tiers
fail by being confidently wrong, and that freedom is what these tests bound.
Three properties carry the weight, each with a concrete failure behind it: a
measured field overwritten by a model would corrupt the impact ranking and the
tuning harness computed from it; an invented path would hand a backend the
wrong file to rewrite, which is the failure the pipeline exists to prevent; and
a session that edited the framework tree would leave the benchmark that follows
measuring an unrecorded change.

Also covers the stage boundary in tracelens_analysis: an unforeseen fault must
cost the audit rather than the run, a restrictive routability hint is honoured
while a permissive one is not, and a harness list the session verified survives
the restamping pass that would otherwise recompute it from the coarse marker
table.

…rded.

A graph replay has no CPU-side parent op, so the profiler records no arguments
for a graph-launched kernel and `shapes` arrives empty. On this model that is
every hot kernel: both 8k/1k runs resolved 0 of 13 and 2 of 12 candidates with
dims. Empty is not a neutral state. With no shapes the tuning backend picks its
own and cannot see the serving configuration, so a prefill kernel serving an
8192-token input was tuned at sequence lengths 512 and 1024, measured 1.1155x,
and lost 3.05% end to end once integrated. The decode kernel of the same pair
guessed close to the real batch and sequence and reported an honest 1.028x.

So the choice was never measured dims against invented ones -- it was which
side invents. The review already reads the model config, the serving arguments
and the run's own artifacts, and the backend reads none of them.

`shapes` and `input_dtypes` therefore leave IMMUTABLE_FIELDS, which now holds
only what cannot be revised for a reason: the trace's own measurements, which
feed the dispatch floor and the closing gain figure, and the keys the row is
joined by. The alternate shape representations move to DERIVED_SHAPE_FIELDS and
are rebuilt, since a harness assembled from a mix of old and new dims still
benchmarks cleanly. Dims carry their own provenance and a session cannot claim
`torch_trace`: collapsing recovered and computed would remove the one signal
that says whether a disappointing integration is worth blaming on the shape.
The forge prompt stops asserting a measurement it no longer has.

Two things had to be fixed for any of it to fire. `apply_revisions` only read
proposal fields on a rewrite that moved the path, so the rows that most need
dims -- the ones the deterministic tiers already located -- would have had
their shapes dropped at the `keep` branch; 10 of this run's 13 candidates take
that path. And review dims had to enter DISPATCHABLE_SHAPE_PROVENANCE, or the
gate would refuse them as untrusted, which is worse than empty: an empty shape
has an override flag and an untrusted provenance has none.

Replayed against the real 13-candidate table, all 7 routable candidates now
carry dims, clear the gate and reach the backend's shape block, against 0
before.

Also drops `recommended_backends` from the prompt. The stamping pass recomputes
it, so inviting a revision there spent the session's effort on a value that was
silently overwritten. And repairs a `_batch_kernel_candidates` monkeypatch left
stale by the gate-rejection change, which the aborted full regressions hid.

Co-authored-by: Cursor <cursoragent@cursor.com>

The correction only matters if it reaches invocation_spec_*.json, which is what
forge reads to construct the test driver. It does: build_invocation_spec takes
`shapes` as the fallback for `input_shapes`, so the review's dims arrive parsed
into invocation.arguments and `inputs` leaves missing_fields. Nothing covered
that link, and it is the one that decides whether the driver benchmarks the
serving workload or a shape the backend chose for itself.

Also pins that only the reviewed table can be resolved as the candidate source.
The pre-review baseline sits in the same directory under kernel_candidates.raw
.json, and resolving that would hand the backend exactly the dims the review
corrected -- the one outcome the two-artifact split exists to prevent.

Co-authored-by: Cursor <cursoragent@cursor.com>

The re-derivation is skipped for candidates that did not change, and the check
asked only whether `source_file` moved. That was complete while a path was the
only thing a revision could alter. It no longer is: operand dims are supplied
most often for a kernel the deterministic tiers already located, so the rows
carrying the hardest-won proposals are precisely the ones the check passed over.

Two analyses in one 8k/1k run staged `review_backfill` dims on `keep` revisions
and shipped a table with none -- the revisions file recorded the proposals and
the candidates never took them. The dims that did land came from rows whose path
happened to move for unrelated reasons.

The condition now names every field a revision can stage. Pinned at the stage
boundary rather than on the re-derivation itself: the unit tests called that
function directly, which is why they passed while the pipeline dropped the data.

Co-authored-by: Cursor <cursoragent@cursor.com>

The review was handed `category_data/` and `priority_data.json` alongside
`analysis.md`. Only the last of those is a supported TraceLens output; the rest
of that directory is internal and the maintainer has said it may be cleared.
Building on it earns a dependency that can disappear between releases.

Almost nothing is given up. For every operator the sidecars describe,
`analysis.md` carries the same operand dims and the same `file(line): function`
launcher in its own table -- the parser already reads both from there, and the
CSV lookups only run once that came back empty. Checked against the current
8k/1k run: 4 of 14 rows carry dims, the same 4 the sidecars can account for, and
for the other 10 neither has anything. A graph replay has no CPU-side parent op,
so nothing recorded those arguments anywhere; withdrawing the sidecars does not
make that worse.

The backfill route survives, re-aimed at the contract. A composite operator that
kept its module attribution names the device kernels it launches in its Kernel
Name cell while carrying the arguments those kernels' own rows lack, so the dims
are recoverable from `analysis.md` alone -- 2 of the 10 empty rows in this run.
The prompt now asks for that, warns that a neighbouring instantiation of the
same kernel family is a different problem size, and tells the session not to go
looking for the intermediate files.

Deterministic-stage readers of the sidecars are untouched here; they are a
larger question than this stage.

Co-authored-by: Cursor <cursoragent@cursor.com>

…tuning.

The phase's own source-level dispatch sat at the end of the GEMM branch, so
`INFERENCE_OPTIMIZER_SKIP_GEMM_TUNING=1` removed it too. The two settings are
unrelated -- one tunes GEMM shape tables, the other rewrites kernel source --
and nothing in the log connected them.

What that cost, in the run that exposed it: the review resolved fourteen hot
kernels to eight routable ones, `reusable_native_kernel_ids` carried all eight,
and k001 at 10.02% cleared the dispatch floor. KERNEL then ran for 65 minutes
and optimized nothing, because with the deterministic dispatch gone the only
remaining path was an orchestration `run_optimization` request, and across the
whole phase the orchestrator sent eight heartbeats, five alerts and no requests
at all. The phase exited on `kernel_no_more_leverage` with every candidate
untried.

Both routes now end in one shared tail: re-profile, the independently gated
fusion and collective stages, then the kernel_opt batch when untried routable
candidates remain. Each stage consults only its own switch, and each skip is a
return inside its own helper rather than out of the entry hook -- which is what
made the GEMM one reach so much further than its name suggested. Verified
across all four combinations of the two skip flags: kernel_opt and collective
fire in every one.

`_should_continue_kernel_after_gemm` and `_run_kernel_opt_after_gemm` are
renamed for the same reason. A name that encodes a dependency the code does not
have is how this hid; what the dispatch needs is untried candidates, so it now
says so. No kernel_id is named at the call site, leaving the set to the
handler's own filter: naming one would put the phase back in the business of
picking, which is the part that stalls when nobody picks.

Also fixes a test that commit 53e3c55 left red: its candidate declares no
gpu_pct, reads as 0 against the 5% floor, and a named gate-rejected kernel now
returns skipped instead of spawning the backend. The evidence flags it asserts
have nothing to do with the floor, so the candidate gets a share that clears it.

Co-authored-by: Cursor <cursoragent@cursor.com>

… shape.

`last_profile_workload` has two writers. The roofline path records it through
`record_profile_workload(task_params)`, which fills `server_args` and
`extra_envs` from those params; the kernel-entry path records it through
`profile_workload_context()` with no params, which leaves both empty. The
staleness check then compared the whole dict against a freshly built no-params
context, so a record written by the first writer could never match.

Every session paid for that once. The warm-recipe replay lands before the
initial roofline -- PRELUDE defers the roofline precisely so the trace reflects
it -- and the optimization stack then holds one entry and gains no more before
KERNEL. Entry still logged "active runtime context changed", re-profiled, and
ran a second TraceLens pass: about fifty minutes to rediscover a configuration
that had not moved. It never fired twice, because the re-profile it forced
rewrote the record in the second writer's shape.

The comparison now projects both sides onto the fields that say what was
profiled -- framework, precision, model, tp, conc, isl, osl, max_model_len --
and leaves out the ones that say how the profile task was parameterized. The
serving configuration keeps its own check, which reads from `current_best` on
both sides and was made symmetric for this same reason.

Verified against the recorded state of the run that exposed it: a
roofline-shaped record reports changed under the old comparison and unchanged
under the new one, a kernel-entry-shaped record reports unchanged under both,
and an actual workload change is still caught.

Co-authored-by: Cursor <cursoragent@cursor.com>

…shape.

The previous commit stopped a re-profile that fired on the recording shape
alone. What it must not have done is stop one that a real configuration change
should force, since a configuration EXPLORE integrated changes which kernels
run and makes an earlier trace genuinely stale.

Parameterized over the cases that separate the two: nothing moved, EXPLORE added
a server argument, EXPLORE added an environment variable, an environment
variable changed value, and the workload itself changed. Only the first is
forgiven. The assertion goes through both checks together, because which of them
catches a given change is an implementation detail -- the property is that the
re-profile happens.

All six fail against the whole-dict comparison, including the four that pass for
the wrong reason there: it reports every case as changed, so it cannot tell a
new configuration from a record it merely did not write itself.

Co-authored-by: Cursor <cursoragent@cursor.com>

A graph-launched kernel has no cpu_op parent, so the profiler strips its
argument dims. The predispatch gate read that absence as a reason to refuse the
dispatch, which does not produce a measured shape -- it only makes the hottest
kernels of a captured model permanently unoptimizable. A run held six routable
candidates, every one of them shapeless, and reached the phase's end having
optimized nothing while the orchestration LLM retried the same rejection.

The backend already covers this: the invocation spec lists the absent operands
under its missing list, and driver preparation recovers them from the kernel
source, the tests the spec names and the deployment context it carries.

Provenance is still validated, but only for a shape that is there. On a
shapeless row the marker names why the dims are absent, and reading that as an
untrusted operand dim would close the same door from the other side.

The escape hatch goes with the gate it bypassed: a flag that can only turn off
something already off is a question every later reader has to answer again.

Co-authored-by: Cursor <cursoragent@cursor.com>

…here.

The dispatch floor and the per-optimization wall clock were both set for a model
whose hot kernels stack up. On a 60-layer sparse-MoE decoder they do not: the six
rewritable candidates of a MiniMax-M3 MXFP4 TP8 run measured 6.9 / 4.9 / 3.7 /
2.6 / 2.5 / 1.9 percent of GPU time, and the operators that outrank them are
vendor binaries with no source to rewrite. So the floor does not order the work,
it decides whether there is any: at 5% exactly one candidate was dispatchable,
the phase spent its attempts re-reaching that same kernel, and the idle guard
wound it down to SWEEP with nothing optimized. 3% makes the addressable set three
kernels; a candidate worth ~4% is worth an attempt when nothing larger exists.

The budget reads as twice what it buys, because forge-loop holds back a finalize
reserve of half the window. At 60 the campaign on that one candidate finished a
single iteration -- planning alone took 16 of its 30 usable minutes -- and
terminated on budget_exhausted, which the ledger then records as a kernel that
could not be improved rather than one that was tried once. 90 leaves ~45 usable
minutes, enough for a second iteration to act on what the first measured.

The request template told the agent to send budget_minutes: 60 verbatim, and it
did, on every request. Since the payload outranks the shipped default, raising
that default alone would have moved only the Coordinator's own dispatch. Drop the
field from the template so one owner sets the clock. The same template also
advertised a 10% floor that no longer matched the shipped 5%, which is how an
orchestration agent came to nominate four sub-floor kernels in a row.

Co-authored-by: Cursor <cursoragent@cursor.com>

The test named for the parity between _batch_kernel_candidates and
SharedState.untried_hot_reusable_kernels never called the second one, so it
could not observe drift at all. What it did assert passed for an unrelated
reason: its two sub-threshold and super-threshold rows shared a source file, so
the op-fanout dedup merged the weaker away and the exclusion held no matter
where the floor sat. It went on asserting a 10% boundary through a 5% default
and then a 3% one without ever failing.

Run both gates and require the same selection. Give every row its own source
file so the dedup cannot stand in for the floor, and assert the recorded skip
reason is below_min_gpu_pct -- without that the shared-source arrangement still
satisfies the equality, which is exactly how this went unnoticed. Derive the
straddling pair from the shipped default; pinning that default's own value
belongs to test_untried_hot_kernels_returns_only_reusable_above_threshold, and
writing it in here is what went stale.

Verified by mutation: lowering one side's floor fails on the parity assertion,
and restoring the shared source file fails on the skip-reason assertion.

Co-authored-by: Cursor <cursoragent@cursor.com>